Image stylization is a process of creating an image according to a particular style, where an input image and a style can be combined so that an output image is the input image produced according to the style. The style can be specified by a set of image analogies, often represented by an input style image and an output style image. By analogies, an image converter will take in the input image and create an output image that is to the input image as the output style image is to the input style image.
[HERTZMANN, A. et al., Proceedings of the 28th annual conference on Computer graphics and interactive techniques, SIGGRAPH '01, 327-340 (2001)] described image stylizations using analogies. Their method begins with a user-supplied image A and its analogy A′. When given an additional input image B, the method computes an output image B′ with the property that visually B′ is to B as A′ is to A. The images A and A′ effectively define a style, and that style is applied to the input image B. Many of the results achieve an appealing hand-painted or drawn appearance.
A wide variety of stylizations can be described by regions of pattern (e.g., watercolor or charcoal texture, cross-hatching, etc.) delineated by some kind of outlines. [BÉNARD, P. et al., Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering 2010) 29, 4 (june), 1497-1506 (2010)] defines three goals they say are necessary for successful temporal coherent stylization of pattern: (1) flatness, (2) coherence motion and (3) temporal continuity. In addition to these three, to achieve the warmth of hand-drawn or hand-pointed imagery in the medium of computer animation, a fourth goal is also to be considered in that outlines should be taken into account.
What might be useful to still image analogies is a counterpart in animation, wherein the input is an animation sequence, i.e., a series of frames that convey motion of objects in a scene. When the input is a sequence of images that convey movement, additional difficulties arise that should be dealt with to result in a pleasing animation sequence.
The concept of image analogies is known, as is the wider field of texture synthesis. However, animation adds new complications. A number of researchers have investigated the use of texture synthesis for creating temporal image sequences, but none have specifically addressed the problems of using texture synthesis to author high-quality non-photorealistic renderings of character animation. [WEI, L.-Y. et al., Eurographics 2009, State of the Art Report, EG-STAR, Eurographics Association (2009)] refers to flow-guided texture synthesis including the synthesis of video exemplars and texturing fluids, which combines texture synthesis with advection, but uses it to address different problems. [BARGTEIL, A. W. et al., Symposium on Computer Animation, 345-351 (2006)] and [KWATRA, V. et al., IEEE Trans. Vis. Comput. Graph. 13, 5, 939-952 (2007)] synthesize temporally coherent textures on the surface of liquids to create surface details such as waves or bubbles and [HAN, J. et al., The Visual Computer 22, 9-11 (2006)] synthesizes animated patterns on the surface of 3D meshes to help visualize flow fields. [WEI, L.-Y., AND LEVOY, M. SIGGRAPH, 479-488 (2000)] shows the synthesis of video textures, but they still do not solve the basis problems.
[HASHIMOTO, R., JOHAN, H., AND NISHITA, T., Computer Graphics International, 312-317 (2003)] applied the image analogies approach to low-resolution video sequences using motion estimation to increase temporal coherence. With the combined problems of inaccurate motion estimation, no explicit handling of occlusion and low overall resolution, the results cannot be described as high-quality.
[BONNEEL, N. et al., 15th International Workshop on Vision, Modeling and Visualization (2010)] used guided texture synthesis to create a system that targets near-interactive pre-visualization of natural environments. The system is well-designed for its target purpose, as it provides the quality needed for pre-visualization using only very approximate geometry while operating within the constraints of near real-time performance. The application of character animation, however, demands a much higher-quality result, and one with better temporal coherence. At the same time, it can accept much slower running times.
There is, of course, a large literature on non-photorealistic stylization techniques that rely on explicit vector representations of lines or strokes. [FREEMAN, W. T., AND ADELSON, E. H., IEEE Trans. Pattern Anal. Mach. Intell. 13, 9, 891-906 (1991)] and [HERTZMANN, A., Proceedings of the 13th Eurographics Workshop on Rendering Techniques, 233-246 (2002)] have investigated example-based stylizations of vector methods. Extracting high-quality time-coherent vector representations from traditional 3D character animation, however, remains a difficult problem, so vector approaches have thus far been used only in limited contexts or with models constructed for particular styles.